Deriving and using document and site quality signals from search query streams

ABSTRACT

A system analyzes one or more search streams to detect one or more associations between a document and one or more queries in the one or more search streams. The system further derives a value for the document based on the detected associations and uses the derived value in evaluating a quality of the document with respect to one or more subsequent queries.

RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.13/112,375, filed May 20, 2011, which is a continuation of U.S.application Ser. No. 11/139,999, filed May 31, 2005, now U.S. Pat. No.7,962,462. The entire disclosures of these applications are incorporatedherein by reference.

BACKGROUND

1. Field of the Invention

Implementations described herein relate generally to informationsearching and, more particularly, to deriving and using document qualitysignals from search query streams.

2. Description of Related Art

Existing information searching systems use search queries to searchthrough aggregated data to retrieve specific information thatcorresponds to the received search queries. Such information searchingsystems may search information based locally, or in distributedlocations. The World Wide Web (“web”) is one example of information indistributed locations. The web contains a vast amount of information,but locating a desired portion of that information can be challenging.This problem is compounded because the amount of information on the web,and the number of new users inexperienced at web searching, are growingrapidly.

Search engines attempt to return hyperlinks to web documents in which auser is interested. Generally, search engines base their determinationof the user's interest on search terms (e.g., in a search query providedby the user). The goal of the search engine is to provide links to highquality, relevant results to the user based on the search query.Typically, the search engine accomplishes this by matching the terms inthe search query to a corpus of pre-stored web documents. Web documentsthat contain the user's search terms are considered “hits” and arereturned to the user.

To return the “best” results of a search, it is important to measure, insome fashion, the quality of documents, such as web documents. Oneexisting document quality measurement technique calculates anInformation Retrieval (IR) score that is a measure of how relevant adocument is to a search query. The IR score can be weighted in variousways. For example, matches in a document's title might be weighted morethan matches in a footer. Similarly, matches in text that is of largerfont or bolded or italicized may be weighted more than matches in normaltext. A document's IR score may be influenced in other ways. Forexample, a document matching all of the terms of the search query mayreceive a higher score than a document matching one of the terms. All ofthese factors can be combined in some manner to generate an IR score fora document that may be used in determining a quality of the results froman executed search.

Scores derived from an existing link-based document ranking algorithmmay additionally be used in conjunction with IR scores. PageRank is oneexisting global, link-based document ranking algorithm that derivesquality signals from the link structure of the web. Often, however, linkstructure may be unavailable, unreliable, or limited in scope, thus,limiting the value of using PageRank in ascertaining the relativequality of some documents.

SUMMARY OF THE INVENTION

According to one aspect consistent with the principles of the invention,a method may include analyzing one or more search streams to detect oneor more associations between a document and one or more queries in theone or more search streams. The method may further include deriving avalue for the document based on the detected associations and using thederived value in evaluating a quality of the document with respect toone or more subsequent queries.

According a further aspect, a method may include using a search query toreturn a set of search result documents and analyzing documents from theset of search result documents to identify a document having prominenttext similar to text of the search query. The method may further includeassigning a value to the identified document and using the assignedvalue as an indication of a quality of the identified document forsubsequently performed searches.

According to another aspect, a method may include using a search queryto return a set of search result documents and identifying links, from acorpus of documents, having text similar to text of the search query,where more of the identified links point to a particular document thanto other documents in the set of search result documents. The method mayfurther include assigning a value to the particular document and usingthe assigned value as an indication of a quality of the particulardocument for subsequently performed searches.

According to a further aspect, a method may include receiving a same, orsimilar, search query from multiple users and retrieving search resultdocuments using the search query. The method may further include sendingthe search result documents to the users and receiving an indication ofa selection of a same document from the search result documents frommore of the multiple users than selected other documents of the searchresult documents. The method may also include assigning a value to thesame document and using the assigned value as an indication of a qualityof the same document for subsequently performed searches.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated in and constitute apart of this specification, illustrate embodiments of the invention and,together with the description, explain the invention. In the drawings,

FIG. 1 is a diagram of an overview of an exemplary use of the invention;

FIG. 2 is a diagram of an exemplary network in which systems and methodsconsistent with the principles of the invention may be implemented;

FIG. 3 is an exemplary diagram of a client and/or server of FIG. 2 in animplementation consistent with the principles of the invention;

FIG. 4 is a flowchart of an exemplary process for assigning points todocuments for use as a quality signal;

FIG. 5 is a diagram illustrating a comparison of prominent text in adocument with a search query to determine if they are similar;

FIG. 6 is a diagram illustrating a comparison of text in links thatpoint to a document with a search query to determine if they aresimilar;

FIG. 7 is a diagram illustrating the selection of a same document from aset of search result documents by multiple users who have issued thesame, or similar, search query to a search engine; and

FIGS. 8-10 illustrate exemplary implementations for determining whethersearch queries “ask” specifically for a document by name.

DETAILED DESCRIPTION

The following detailed description of the invention refers to theaccompanying drawings. The same reference numbers in different drawingsmay identify the same or similar elements. Also, the following detaileddescription does not limit the invention.

According to aspects of the invention, methods and systems are providedthat determine a quality signal for a document, or corresponding site,based on whether search queries specifically “ask” for or request thedocument. Consistent with aspects of the invention, whenever users issuea search query for which they already have a desired document/site inmind, then issuance of the search query may be considered a strong“vote” for the quality of the document/site.

A “document,” as the term is used herein, is to be broadly interpretedto include any machine-readable and machine-storable work product. Adocument may include an e-mail, a web site, a file, one or more digitalimages, a combination of files, one or more files with embedded links toother files, a news group posting, a blog, a web advertisement, etc. Inthe context of the Internet, a common document is a web page. Web pagesoften include textual information and may include embedded information(such as meta information, images, hyperlinks, etc.) and/or embeddedinstructions (such as Javascript, etc.). A “link,” as the term is usedherein, is to be broadly interpreted to include any reference to or froma document.

Exemplary Overview

FIG. 1 illustrates an overview of the determination and use of a qualitysignal to be associated with one or more documents. In accordance withthis aspect of the invention, a search query, and, possibly, userselections from search results resulting from execution of the searchquery, may be analyzed to determine if the user was requesting aspecific document or site from a corpus of documents or sites.

As shown in FIG. 1, a user may issue a search query 105 to a searchengine. The search engine may search a corpus of documents 110, usingexisting searching techniques, based on search query 105. As a result ofthe search, the search engine may return indications (e.g., links and/ordescriptions) of a set of search result documents 115 to the user whoissued search query 105. Points may be assigned 130 by the search engineto one or more documents 120 of search result documents 115 if searchquery 105 is deemed to “ask” for the one or more specific documents 120.Search query 105 may be determined to “ask” for the one or moredocuments 120 if each of the documents has similar text (e.g., similartext in the title of the document, in prominent text in the document, orin a URL of the document) as compared to the text of the search query.Search query 105 may be determined to “ask” for the one or moredocuments 120 if more links (e.g., a majority) contained in otherdocuments of the corpus of documents 110, having text that is similar tothe text of the search query, point to the one or more documents 120than to other documents in the search result documents 115.

In some implementations, the user may select one or more documents 125from search result documents 115. Points may be assigned 135 by thesearch engine to the user selected document(s) 125 if search query 105“asks” for the document(s). The search query 105 may be determined to“ask” for the document(s) 125 if many previous users who have issuedthis particular search query 105 selected document(s) 125 from a set ofsearch result documents.

The points assigned to documents contained in the corpus of documents110 may then be used 140 as a quality signal for each respectivedocument for subsequent searches performed by a search engine. In oneimplementation, the assigned points may be used as a quality signal forall subsequent searches and, thus, may serve as a “global” qualitysignal for subsequent searches regardless of whether the subsequentsearches include the same search query 105 that resulted in points beingassigned to a specific document.

The assigned points may be used, for example, in any type ofsubsequently executed document scoring/ranking algorithm. In oneimplementation, the assigned points may be used as an input in asubsequent PageRank computation. In another implementation, acombination of the points assigned consistent with aspects of theinvention and results of a PageRank computation may be used to derive aquality signal. This combination may either be mathematical (e.g., anaverage) or otherwise (e.g., using different signals at different placesor times in a ranking/scoring algorithm). The points assigned,consistent with aspects of the invention, may generally be used as asignal of document quality, and can be used in many different ways inany scoring/ranking algorithm, or for deriving other quality signalsthat are used in a ranking/scoring algorithm.

Exemplary Network Configuration

FIG. 2 is an exemplary diagram of a network 200 in which systems andmethods consistent with the principles of the invention may beimplemented. Network 200 may include multiple clients 210 connected tomultiple servers 220 and 215 via a network 230. Network 230 may includea local area network (LAN), a wide area network (WAN), a telephonenetwork, such as the Public Switched Telephone Network (PSTN), anintranet, the Internet, a memory device, or a combination of networks.Two clients 210 and two servers 220 and 215 have been illustrated asconnected to network 230 for simplicity. In practice, there may be moreor fewer clients and servers. Also, in some instances, a client mayperform the functions of a server and a server may perform the functionsof a client.

Clients 210 may include client entities. A client entity may be definedas a device, such as a wireless telephone, a personal computer, apersonal digital assistant (PDA), a laptop, or another type ofcomputation or communication device, a thread or process running on oneof these devices, and/or an object executable by one of these devices.Servers 220 and 215 may include server entities that gather, process,search, and/or maintain documents in a manner consistent with theprinciples of the invention. Clients 210 and servers 220 and 215 mayconnect to network 230 via wired, wireless, and/or optical connections.

In an implementation consistent with the principles of the invention,server 220 may include a search engine 225 usable by users at clients210. Server 220 may implement a data aggregation service by crawling acorpus of documents (e.g., web pages) hosted on data server(s) 215 andstore information associated with these documents in a repository ofcrawled documents. The data aggregation service may be implemented inother ways, such as by agreement with the operator(s) of data server(s)215 to distribute their hosted documents via the data aggregationservice. Search engine 225 may execute a query, received from a user, onthe corpus of documents hosted on data server(s) 215.

Server(s) 215 may store or maintain documents that may be crawled byserver 220. Such documents may include data related to published newsstories, products, images, user groups, geographic areas, or any othertype of data. For example, server(s) 215 may store or maintain newsstories from any type of news source, such as, for example, theWashington Post, the New York Times, Time magazine, or Newsweek. Asanother example, server(s) 215 may store or maintain data related tospecific product data, such as product data provided by one or moreproduct manufacturers. As yet another example, server(s) 215 may storeor maintain data related to other types of web documents, such as pagesof web sites.

While servers 220 and 215 are shown as separate entities, it may bepossible for one or more of servers 220 and 215 to perform one or moreof the functions of another one or more of servers 220 and 215. Forexample, it may be possible that two or more of servers 220 and 215 areimplemented as a single server. It may also be possible for a single oneof servers 220 or 215 to be implemented as two or more separate (andpossibly distributed) devices.

Exemplary Client/Server Architecture

FIG. 3 is an exemplary diagram of a client or server entity (hereinaftercalled “client/server entity”), which may correspond to one or more ofclients 210 and servers 220 and 215, according to an implementationconsistent with the principles of the invention. The client/serverentity may include a bus 310, a processing unit 320, an optional mainmemory 330, a read only memory (ROM) 340, a storage device 350, an inputdevice 360, an output device 370, and a communication interface 380. Bus310 may include a path that permits communication among the componentsof the client/server entity.

Processing unit 320 may include any type of software, firmware orhardware implemented processing device, such as, a microprocessor, afield programmable gate array (FPGA), combinational logic, etc. Mainmemory 330 may include a random access memory (RAM) or another type ofdynamic storage device that stores information and instructions forexecution by processing unit 320, if processing unit 320 includes amicroprocessor. ROM 340 may include a conventional ROM device or anothertype of static storage device that stores static information and/orinstructions for use by processing unit 320. Storage device 350 mayinclude a magnetic and/or optical recording medium and its correspondingdrive.

Input device 360 may include a conventional mechanism that permits anoperator to input information to the client/server entity, such as akeyboard, a mouse, a pen, voice recognition and/or other biometricmechanisms, etc. Output device 370 may include a conventional mechanismthat outputs information to the operator, including a display, aprinter, a speaker, etc. Communication interface 380 may include anytransceiver-like mechanism that enables the client/server entity tocommunicate with other devices and/or systems. For example,communication interface 380 may include mechanisms for communicatingwith another device or system via a network, such as network 230.

As will be described in detail below, the client/server entity,consistent with the principles of the invention, may perform certainsearching-related operations. The client/server entity may, in someimplementations, perform these operations in response to processing unit320 executing software instructions contained in a computer-readablemedium, such as memory 330. A computer-readable medium may be defined asone or more physical or logical memory devices and/or carrier waves.

The software instructions may be read into memory 330 from anothercomputer-readable medium, such as data storage device 350, or fromanother device via communication interface 380. The softwareinstructions contained in memory 330 may cause processing unit 320 toperform processes that will be described later. Alternatively, hardwiredcircuitry may be used in place of, or in combination with, softwareinstructions to implement processes consistent with the principles ofthe invention. Thus, implementations consistent with principles of theinvention are not limited to any specific combination of hardwarecircuitry and software.

Exemplary Quality Signal Determination Process

FIG. 4 is a flowchart of an exemplary process for determining a qualitysignal associated with documents from a corpus of documents consistentwith principles of the invention. As one skilled in the art willappreciate, the process exemplified by FIG. 4 can be implemented insoftware and stored on a computer-readable memory, such as main memory330, ROM 340 or storage device 350 of server 220. In otherimplementations, the processing exemplified by FIG. 4 can be implementedin hardwired circuitry, such as combinational logic, within processingunit 320 of server 220.

The exemplary process may begin with the receipt of a search query (act405). Search engine 225 at server 220 may receive the search query froma user associated with a client 210. Alternatively, the search query maybe retrieved from a log of previously issued user search queries. Searchengine 225 may search a corpus of documents using the received searchquery (act 410) using existing searching techniques. Search engine 225may then send the results of the search to the user who issued thesearch query (act 415). The results sent to the user may includeindications (e.g., links and/or descriptions) of a set of search resultdocuments that were returned as a result of the search of the corpus ofdocuments.

Search engine 225 may assign points to one or more documents from thesearch results if the received search query is deemed to have “asked”for the document(s) (act 420). A user may thus “vote” in favor of thequality of the requested documents (i.e., the documents which the useris deemed to have asked for) through issuance of the search query. Thenumber of points assigned to each document may be predetermined (e.g.,one point), or may vary based on various factors. In someimplementations, receipt of certain pre-designated search queries mayresult in no assignment of points to the one or more documents containedin the search result documents. Additionally, in other implementations,the assigned points may include negative points. For example, if it isknown that users issuing a specific query are issuing it solely toattempt to amass points for a specific document, then points may besubtracted from points assigned to the document.

A number of techniques can be used to determine whether a given searchquery “asks” for one or more documents. For example, the received searchquery may be determined to “ask” for the one or more documents if eachof the documents is analyzed and determined to have similar text (e.g.,similar text in the title of the document, in prominent text in thedocument, or in the document's uniform resource locator (URL)) ascompared to the text of the search query. As an illustrative example,FIG. 5 depicts a document 505, having a body of text 510 that includesprominent text (e.g., “similar text”) that is similar to the text in asearch query 515. Search query 515 may thus be considered to “ask” fordocument 505 based on the similar text 510 prominently featured indocument 505. Text prominently featured in document 505 may include, forexample, text in the document title, text that has a large font size,text in bold, text that includes all capital letters, or text positionedat certain locations in the document (e.g., at the top of the document).

As another example, the received search query may be determined to “ask”for the one or more documents if more links (e.g., a majority, or apreponderance, of links) contained in other documents of the corpus ofdocuments, having text that is similar to the text of the search query,point to the one or more documents than to other documents of the searchresult documents. For example, assume that the queries [London Hotels]and [Ritz Carlton] are frequently issued by users. If a preponderance oflinks containing the text “Ritz Carlton” point to an official RitzCarlton hotel document, it can be determined that users were asking forthe official Ritz Carlton document. Whereas, if no document in thecorpus of documents has a preponderance of links with the text “LondonHotels” pointing to it, then it may be determined that users were notasking for any particular one of those documents, even though some ofthose documents might have more links pointing to it with the text“London Hotels” than the Ritz Carlton document had links pointing to itwith the text “Ritz Carlton.” In such a case, the official Ritz Carltondocument may receive points from the query [Ritz Carlton], but nodocument may receive points from the query [London Hotels].

As an illustrative example, FIG. 6 depicts linking documents 605 and 610that each have links 615 and 620, respectively. Link 615 points todocument 625 and has text (i.e., anchor text) that is similar to thetext of search query 630, and link 620 points to document 625 and hastext that is similar to the text of search query 630. Search query 630may, thus, be considered to “ask” for document 625 based on the similartext contained in links 615 and 620 pointing to document 625. In oneimplementation, a threshold number of links (e.g., greater than 20links), with similar text to the search query, that points to the one ormore documents may be required before determining that the search query“asks” for the one or more documents. In another implementation, amajority of links, having similar text to the search query, that pointto the one or more documents may be required before determining that thesearch query “ask” for the one or more documents.

A user selection(s) from the search result documents, sent in act 415,may be received by server 220. For example, a user at client 210 whoreceived an indication of the search results from server 220 may select(e.g., “click” on a link) of one or more search result documents. As athird example, the received search query may then be determined to “ask”for the one or more documents if the user selects the one or moredocuments from the set of result documents and many previous users(e.g., a specified minimum number, specified percentage, orpreponderance, of users) who have issued the same, or similar, searchquery, selected those documents as well. As an illustrative example,FIG. 7 depicts the issuance of a same, or similar, search query 705 by Nusers 710-1 through 710-N. A search engine (not shown) retrieves one ormore search result documents 715 from a corpus of documents usingconventional searching techniques based on search query 705. Indicationsof the search result documents 715 may be sent to each of users 710, andeach user 710 may “select” a document 720 from the search resultdocuments 715. As shown in FIG. 7, each user 710 has selected the samedocument 720 from search result documents 715, thus, effectivelyindicating that each user 710 has asked for document 720 by issuingsearch query 705.

If an analysis of all instances of a given query (e.g., query 705), frommany users, indicates that no one search result document was selectedmost of the time (e.g., at least 51 of 100 selections by users fromsearch result documents), then it can be determined that the users werenot “asking” for a specific document by issuing a same search query. Forexample, assume that the query [London Hotels] is issued by users veryfrequently and the query [Ritz Carlton] is issued by users lessfrequently. However, over all the search queries for [London Hotels]issued by many users, not a single search result garners a majority ofselections by the users. This, thus, suggests that there was no oneparticular document that the users were “asking” for when they issuedsearch query [London Hotels]. So even though certain documents in thesearch results from the query [London Hotels] received many userselections, those documents may not be assigned any points for thesearch query [London Hotels] (though some of these documents may beassigned points for other search queries). For the query [Ritz Carlton],however, the official Ritz Carlton document may receive a vast majorityof all user selections and may thus be assigned points as the documentfor which the users were “asking,” despite receiving fewer user totalselections than for many of the search result documents resulting fromthe search query [London Hotels].

The points assigned to respective documents may be used as a qualitysignal in subsequently performed searches (act 425). The subsequentlyperformed searches may be performed by search engine 225, or by othersearch engines. In one implementation, the assigned points may be usedas a quality signal for all subsequent searches, thus, serving as a“global” quality signal for subsequent searches regardless of whetherthe searches include the same search query that resulted in points beingassigned to a specific document. A document that has been assignedpoints in act 430 may, therefore, get a “boost” (e.g., be scored/rankedhigher) for other search queries that match the content of the document.Acts 405-430 may be repeated for each search query received from a userat a client 210.

The points assigned in act 425 above can be limited, or modified, in anumber of ways. For example, the number of points that can be assignedto a document from the receipt of a same query may be limited (e.g., nomore than 50 points for the search query “xxxxx”), the number of pointsdue to issuance of search queries from a single user may be limited(e.g., a maximum of 100 points for a single user, assuming 1 pointassigned per search query), the number of points a user can give aparticular document may be limited (e.g., no more than 20 points fordocument X from user Y), or the number of points due to issuance ofsearch queries from a same Internet Protocol (IP) address may be limited(e.g., no more than 30 points from IP address mmmm.nnnn.oooo.pppp). Anyof the above ways for limiting, or otherwise modifying, assigned pointsmay be limited by time (e.g., no more than 50 points for the searchquery “xxxx” during one day, no more than 20 points for document X fromuser Y during a week). Any, or all of, the above-described pointassignment, modification or limitation, techniques may be combined inmultiple different ways, and a mathematical function may be used tocompute a quality signal for a given document.

Exemplary Implementations

FIG. 8 illustrates an exemplary implementation in which similaritybetween text in a URL 810 associated with a document 805 and multiple,similar search queries 820 is used to determine whether the searchqueries 820 are asking for a document 805 by name. FIG. 8 depictsdocument 805 having a URL 810 “mail.yahoo.com.” A portion 815 of text ofURL 810 is similar to the text contained in each of the multiple searchqueries 820, which all have search terms “yahoo” and “mail.” Based onthe similarity between the text in URL 810 and each of the searchqueries 820, a determination can be made, consistent with one aspect ofthe invention, that search queries 820 are each “asking” for document805.

FIG. 9 illustrates a further exemplary implementation in whichsimilarity between text, contained in links, more of which point to adocument in a set of search results than to other documents in the setof search results, and a search query is used to determine whether thesearch query is asking for a document. As illustrated in FIG. 9, adocument 905 may have a URL 910 “www.bostonredsox.com.” Four linkdocuments 915-1 through 915-4 may each include a link, pointing todocument 905, that include similar text (e.g., “red sox”) to a searchquery 925. Based on the number of links in documents 915 pointing todocument 905 (with no links to other documents in the set of searchresult documents), and having similar text to the search query, adetermination can be made, consistent with an aspect of the invention,that search query 925 is “asking” for document 905 by name.

FIG. 10 illustrates another exemplary implementation in which it isdetermined that a document is effectively “asked” for when many users(e.g., a majority, or preponderance, of users) who issue a search queryselect the same document from a set of search results. FIG. 10 depictsthe issuance of a same, or similar, search query 1005 (“yahoo mail”) byN users 1010-1 through 1010-N. A search engine (not shown) retrieves oneor more search result documents 1015 from a corpus of documents usingconventional searching techniques based on search query 1005.Indications of the search result documents 1015 may be sent to each ofthe users 1010, and a set of users 1010 “select” a same document 1020(“www.mail.yahoo.com”) from search result documents 1015. Since a set ofusers 1010 have selected the same document 1020 from search resultdocuments 1015, search query 1005 is effectively “asking” for document1020. Other users (not shown) issuing search query 1005 may selectdifferent documents from search result documents 1015, as long as thenumber of user selections of document 1020 is sufficiently high (e.g., amajority) as compared to user selections of the other differentdocuments. In one implementation, the selection of a same document 1020by a set of users 1010 in response to issuance of a same, or similar,search query 1005 may be used to increase a quality score associatedwith document 1020. Thus, even if subsequent users issue a differentsearch query than search query 1005 (e.g., “web mail”), then document1020 may be ranked higher in the resulting search results due to thisincreased quality score.

CONCLUSION

The foregoing description of preferred embodiments of the presentinvention provides illustration and description, but is not intended tobe exhaustive or to limit the invention to the precise form disclosed.Modifications and variations are possible in light of the aboveteachings or may be acquired from practice of the invention. Forexample, while a series of acts has been described with regard to FIG.4, the order of the acts may be modified in other implementationsconsistent with the principles of the invention. Additionally,non-dependent acts may be performed in parallel.

While aspects of the invention have been described as assigning pointsto documents for use as a quality signal, points may additionally, oralternatively, be assigned to a “site” that hosts the various documents.Such sites may include a group of documents under common control, suchas a group of documents associated with an organization, a domain name,a host name, or a set of documents created by the same person or groupof persons. A “site” may also include a group of documents about atopic, a group of documents in a particular language, a group ofdocuments hosted in a particular country, or a group of documentswritten in a particular writing style. In all of these variationsinvolving a “site,” a point assigned to a document becomes a vote forthe site with which the document is associated. In other aspects,quality signals may be derived at a combination of levels (e.g.,document, various “site” levels) and combining them in some manner, suchas, for example, during scoring/ranking of the documents.

It will also be apparent to one of ordinary skill in the art thataspects of the invention, as described above, may be implemented in manydifferent forms of software, firmware, and hardware in theimplementations illustrated in the figures. The actual software code orspecialized control hardware used to implement aspects consistent withthe principles of the invention is not limiting of the presentinvention. Thus, the operation and behavior of the aspects of theinvention were described without reference to the specific softwarecode—it being understood that one of ordinary skill in the art would beable to design software and control hardware to implement the aspectsbased on the description herein.

Further, certain portions of the invention have been described as“logic” that performs one or more functions. This logic may includehardware, such as an application specific integrated circuit or a fieldprogrammable gate array, software, or a combination of hardware andsoftware.

No element, act, or instruction used in the present application shouldbe construed as critical or essential to the invention unless explicitlydescribed as such. Also, as used herein, the article “a” is intended toinclude one or more items. Where only one item is intended, the term“one” or similar language is used. Further, the phrase “based on” isintended to mean “based, at least in part, on” unless explicitly statedotherwise.

What is claimed is:
 1. A non-transitory computer-readable memory devicestoring instructions, the instructions comprising: one or moreinstructions which, when executed by at least one processor, cause theat least one processor to: receive a same search query from a pluralityof client devices; identify a plurality of search result documents basedon the search query; provide information regarding the plurality ofsearch result documents to the plurality of client devices; receiveinformation regarding selections, of the information regarding theplurality of search result documents, at the plurality of client devicesin connection with the search query; determine that a particularquantity of the selections, for information regarding a particularsearch result document of the plurality of search result documents, isgreater than a quantity of the selections for information regarding oneor more other search result documents of the plurality of search resultdocuments; assign a value to the particular search result document basedon determining that the particular quantity of the selections is greaterthan the quantity of the selections for the information regarding theone or more other search result documents, the value, assigned to theparticular search result document, being based on: a particular periodof time, or a respective network address associated with one or more ofthe plurality of client devices; and use the value as a measure ofquality for the particular search result document for a subsequentlyreceived search query that is associated with the search query.
 2. Thenon-transitory computer-readable memory device of claim 1, where thevalue, assigned to the particular search result document, is limited foreach client device of the plurality of client devices.
 3. Thenon-transitory computer-readable memory device of claim 1, where theparticular quantity of the selections includes a percentage.
 4. Thenon-transitory computer-readable memory device of claim 1, where the oneor more instructions which cause the at least one processor to use thevalue as the measure of quality comprise: one or more instructionswhich, when executed by the at least one processor, cause the at leastone processor to use the value as one factor of a plurality of factorsto determine the measure of quality for the particular search resultdocument for the subsequently received search query.
 5. Thenon-transitory computer-readable memory device of claim 1, where the oneor more instructions which cause the at least one processor to assignthe value to the particular search result document comprise: one or moreinstructions which, when executed by the at least one processor, causethe at least one processor to assign the value to one or more sitesassociated with the particular search result document.
 6. Thenon-transitory computer-readable memory device of claim 1, where the oneor more instructions to use the value as the measure of quality for theparticular search result document include: one or more instructionswhich, when executed by the at least one processor, cause the at leastone processor to rank the particular search result document higher thananother search result document, in a subsequently performed documentscoring or ranking algorithm, based on the assigned value being higherthan a value assigned to the other search result document.
 7. Thenon-transitory computer-readable memory device of claim 1, where theparticular quantity of the selections corresponds to a majority of theselections.
 8. A system comprising: an interface to receive a searchquery from a plurality of client devices; and a processing unit,implemented at least partially in hardware, to: identify a plurality ofsearch result documents based on a search query, provide informationregarding the plurality of search result documents to the plurality ofclient devices, receive information regarding selections, of theinformation regarding the plurality of search result documents, at theplurality of client devices in connection with the search query,determine that a particular quantity of the selections, for informationregarding a particular search result document of the plurality of searchresult documents, is greater than a quantity of the selections forinformation regarding one or more other search result documents of theplurality of search result documents, assign a value to the particularsearch result document based on determining that the particular quantityof the selections, for the information regarding the particular searchresult document, is greater than the quantity of the selections for theinformation regarding the one or more other search result documents, thevalue, assigned to the particular search result document, being basedon: a particular period of time, and a respective network addressassociated with one or more of the plurality of client devices, and usethe assigned value as a measure of quality for the particular searchresult document for a subsequent search, the subsequent search beingassociated with the search query.
 9. The system of claim 8, where, whenusing the assigned value as the measure of quality for the particularsearch result document for the subsequent search, the processing unit isto: rank the particular search result document higher than anothersearch result document in a subsequently performed document scoring orranking algorithm based on the assigned value being higher than a valueassigned to the other search result document.
 10. The system of claim 8,where, when assigning the value to the particular search resultdocument, the processing unit is further to: assign the value to one ormore sites associated with the particular search result document. 11.The system of claim 8, where the particular quantity of the selectionscorresponds to a majority of the selections.
 12. The system of claim 8,where, when using the assigned value as the measure of quality for theparticular search result document for the subsequent search, theprocessing unit is to: use the value as one factor of a plurality offactors to determine the measure of quality for the particular searchresult document for the subsequent search.
 13. A method comprising:receiving, at one or more processors, a same search query from aplurality of client devices; identifying, by the one or more processors,search result documents based on the search query; providing, by the oneor more processors, information regarding the search result documents tothe plurality of client devices; receiving, at the one or moreprocessors, information regarding selections, of the informationregarding the search result documents, at the plurality of clientdevices in connection with the search query; determining, by the one ormore processors, that a particular quantity of the selections, forinformation regarding a particular search result document of the searchresult documents, is greater than a quantity of the selections forinformation regarding one or more other search result documents of thesearch result documents; assigning, by the one or more processors, avalue to the particular search result document based on determining thatthe particular quantity of the selections, for the information regardingthe particular search result document, is greater than the quantity ofthe selections for the information regarding the one or more othersearch result documents, the value, assigned to the particular searchresult document, being based on at least one of: a particular period oftime, or a respective network address associated with one or more of theplurality of client devices; and using, by the one or more processors,the value as a measure of quality for the particular search resultdocument for a subsequently received search query that is associatedwith the search query.
 14. The method of claim 13, where the particularquantity of the selections includes a percentage.
 15. The method ofclaim 13, where using the value as the measure of quality includes:using the value as one factor of a plurality of factors to determine themeasure of quality for the particular search result document for thesubsequently received search query.
 16. The method of claim 13, whereassigning the value to the particular search result document includes:assigning the value to one or more sites associated with the particularsearch result document.
 17. The method of claim 13, where using thevalue as the measure of quality for the particular search resultdocument includes: ranking the particular search result document higherthan another search result document, in a subsequently performeddocument scoring or ranking algorithm, based on the assigned value beinghigher than a value assigned to the other search result document. 18.The method of claim 13, where the particular quantity of the selectionscorresponds to a majority of the selections.